In order to address the problem that the mutation in the current fuzzing has certain blindness and the samples generated by the mutation mostly pass through the same high-frequency paths, a binary fuzzing method based on light-weight program analysis technology was proposed and implemented. Firstly, the target binary program was statically analyzed to filter out the comparison instructions which hinder the sample files from penetrating deeply into the program during the fuzzing process. Secondly, the target binary program was instrumented to obtain the specific values of the operands in the comparison instructions, according to which the real-time comparison progress information for each comparison instruction was established, and the importance of each sample was measured according to the comparison progress information. Thirdly, the real-time path coverage information in the fuzzing process was used to increase the probability that the samples passing through rare paths were selected to be mutated. Finally, the input files were directed and mutated by the comparison progress information combining with a heuristic strategy to improve the efficiency of generating valid inputs that could bypass the comparison checks in the program. The experimental results show that the proposed method is better than the current binary fuzzing tool AFL-Dyninst both in finding crashes and discovering new paths.

The deobfuscation result of deobfuscation framework Miasm is a picture, which cannot be decompiled to recovery program source code. After deep research on the obfuscation strategy of Obfuscator Low Level Virtual Machine (OLLVM) and Miasm deobfuscation idea, a general OLLVM automatic deobfuscation framework based on symbolic execution was proposed and implemented. Firstly, the basic block identification algorithm was used to find useful basic blocks and useless blocks in the obfuscated program. Secondly, the symbolic execution technology was used to determine the topological relations among useful blocks. Then, the instruction repairment was directly applied to the assembly code of basic blocks. Finally, an executable file after deobfuscation was obtained. The experimental results show that, under the premise of guaranteeing the deobfuscation time as little as possible, the code similarity between the deobfuscation program and the non-obfuscated source program is 96.7%. The proposed framework can realize the OLLVM deobfuscation of the C/C ++ files under the x86 architecture very well.

Based on the systematic study and research on the existing DEX packing and unpacking technologies, a DEX unpacking scheme based on Android ART Virtual Machine (VM) was proposed and implemented. The method could extract the original DEX file from the enhanced Android application. The core idea is to accomplish the zero-knowledge unpacking in a strong compatible way by combining simulation execution with static instrumentation. Firstly, the unpacking point was achieved by inserting monitoring codes into the interpreter of ART. Then, the memory location of the data belonging to original DEX file was obtained by performing simulation execution and analyzing related structs. Finally, the original DEX file was restored by collecting and reassembling the data according to the format of DEX file. The experimental results indicate that the proposed automatically unpacking method can well perform zero-knowledge unpacking by just bringing in little time delay when application launching.

Concerning the fact that when the receiver of an Intent does not validate empty data and abnormal data, the process will crash and cause denial of service, an automated Android component vulnerability mining framework based on static analysis techniques and fuzzing test techniques was proposed. In this framework, reverse analysis techniques and static data flow analysis techniques were used to extract package name, component, Intent with the data of a traffic and data flow paths from exported component to private component to assist fuzzing test. In addition, more mutation strategy on the attributes of Intent (such as Action, Category, Data and Extra) were added while generating Intent tests and the Accessibility technology was adopted to close the crash windows in order to realize automation. Finally, a tool named DroidRVMS was implemented, and a comparative experiment with Intent Fuzzer was designed to verify the validity of the framework. The experimental results show that DroidRVMS can find denial of service vulnerability resulting from dynamic broadcast receiver and most types of exceptions.